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ABSTRACT 



This report describes a computer-monitoring study of users of The Ohio State 
University Libraries 1 online catalog, an established and heavily used 
Information retrieval system* To our knowledge, this 1s the first monitoring 
study of an online catalog performed without system-defined user sessions* 
Online catalogs represent a class of retrieval systems which are designed for 
end users, require little or no formal training, and replace an existing 
manual system* The study characterizes user behavior 1n terms of types of 
searches done, patterns of use, time spent on searching, errors, and system 
problems. Preliminary results suggest that users have much shorter sessions 
on online-catalog systems than on other types of retrieval systems* Patterns 
of use vary between campus libraries, academic quarters, and between short and 
long sessions. Results of the study will be applied to improving the user 
Interface and other system features* 
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1.0 INTRODUCTION 



Information retrieval systems began as a technological innovation which 
provided an alternative means of access to vast amounts 3t data. These highly 
specialized systems required extensive training to use and were usually 
operated by trained search Intermediaries, rather than by the end users who 
needed the Information. 

We are now seeing a major s.h1ft 1n user population. Information retrieval 
systems are rapidly cultivating an end user market 1r variety of domains. 
Time-sharing systems serving home computer jsers, such as CompuServe and The 
Source, have been offering limited Information retrieval services to their 
customers for several years. Videotex services accessible through home 
television sets (occasionally as part of a cable service) offer retrieval from 
news, shopping, and other consumer-oriented database, Two of the major 
bibliographic retrieval services, DIALOG and BRS, have recently announced 
low-cost evening rates for database access with a simplified command language. 
This service 1s also aimed at the personal -computer owner. 

Libraries have been major users and suppliers of Information retrieval systems 
for many years, both 1n the technical and public service areas. Libraries 
have recently joined the end user market by replacing their card catalogs with 
online catalogs. 

We have chosen to study Information retrieval behavior 1n the domain of online 
catalogs both In service to our library audience and because online catalogs 
are a rich test bed for behavioral research 1n human-computer Interaction. 
They offer us an opportunity to study the behavior of people who are motivated 
to seek the Information provided by the system, but who may not be motivated 
to use the technology Itself. The circumstances of online catalog use are 
such that no formal training can be required, and only minimal training can be 
offered, such as a brief pamphlet, a computer-ass1sted-1nstruct1on program, 
Information displays, and reference manuals. Any study of user behavior on a 
system which requires no training 1s Inherently a study of the user Interface 
an<* Its ability to convey Information. 

As libraries began to Implement online catalogs, and many more libraries were 
faced with the decision to adopt these catalogs, the need for more Information 
about online catalog use became apparent. To fill this need, the Council on 
Library Resources funded a nationwide study of online catalogs In public, 
academic, and college libraries (Ferguson et al. 1982; Kaske and Sanders 1983; 
Markey 1983; Tolle 1983). The study, known as the Online Public Access 
Catalog Evaluation Project, examined 17 online catalog Installations at 29 
Institutions in the United States, using multiple research methods. 
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Online catalog users and nonusers were surveyed for their attitudes about the 
systems, library staff were Interviewed, online-monitoring studies were 
performed and the software features were analyzed (Borgman 1982; Tolle 1983; 
Hlldreth 1982). 

This report describes the onl1ne-mon1 torlng study of one online catalog t The 
Ohio State University Library Control System (LCS). Me chose to study the 
Library Control System first and 1n the most depth for several reasons. 
First, LCS, being one of the oldest and most heavily used systems currently 
available, allowed us to study a stable system with a 1 rge population of 
regular users. Second, the system was regularly mon1t< red as part of the 
maintenance process, providing a very large sample of users, across multiple 
campus libraries, and nver a one-year period. Third, we were able to 
corroborate the monitoring data with observation data from another study, the 
NSF-sponsored "Terminal Requirements for Online Catalogs 1n Libraries," 
allowing us to parse the data Into Individual user sessions. These three 
factors combined to allow us to create a much richer dataset than we could 
obtain from other online catalog systems. 
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2.0 RELATED RESEARCH 



The online-monitoring methodology has been applied to user behavior on several 
other Information retrieval systems, but to our knowledge, this study and that 
of Tolle are the first applications of the methodology to online catalogs 
(Tolle 1983). Pennlman and Domlnlck (1980) provide a thorough review of 
Information retrieval -monitoring studies through 1980. While they do not 
explicitly list results for these studies, they, and others, do outline some 
of the variables which were studied: Interactions per session, Interactions 
per minute (Pennlman and Perry 1976); query complexity (Mlttman and Domlnlck 
1973); probability of session length In minutes for different database types 
(Pennlman and Perry 1976); learning curve for terminal users (Kennedy 1975); 
arid most likely paths of next actions In a search session (Pennlman 1975a, 
1975b). 

In his dissertation (1975), Pennlman described users of the BASIS System In 
terms of a Markov model and found that the data supported the model. In a 
•sore recent study (1981), Pennlman compared patterns of use for Infreauent, 
moderate, and frequent users of the MEDLINE System. He found significant 
differences among groups In frequency «f Individual commands. Several of the 
more advanced features of the system were used almost exclusively by the 
frequent users. Infrequent users were slower, while experienced searchers use 
more commands and more connect time than Inexperienced searchers. 

The MEDLINE System has 50 user commands, representing 16 "states" or phases of 
the search, such as logon, term selection, strategy formation, reviewing 
results, and so on. Pennlman found that an average session contained 47.5 
transaction; and lasted 9 minute? ~d 45 seconds, for an average of 4.8 
transacts*: per minute. Penni '*o found chat users had difficulty 
logging onto the system; 69X r ..nlque ID numbers were spurious and 
generated by users attempting , v gon with a mistyped or Incorrect ID number. 
We will compare these results to the LCS data. 

Another use o" monitoring date Is to corroborate survey data about system use. 
Considerable different s may be found between the way In which users say they 
use a system and the tvldence provided by a log of their transactions. In a 
public-funded videotex evaluation, the University of Kentucky/USDA Green Thumb 
System provided system data which correlated less than 301 with reported usage 
figures while showing a clear downward trend over the 13 months recorded 
(Warner and Clearfield 1981; Rice and Paisley 1982). USDA policy and 
marketing Implications based upon system-monitored system data ran counter to 
some of the Implications based upon respondents' reports of their usage. 
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3.0 RESEARCH QUESTIONS 



The purpose of this study was to examine the behavior of system users by 
unobtrusive measures through the use of monitoring data. The 
computer-monitoring methodology was chosen as 1t provides a complete and 
accurate record of online behavior. Computer monitoring of an online system 
1s simply logging (automatically) every transaction made from a terminal and 
the time when 1t occurred (R1ce and Borgman 1983; Borgman and R1ce 1982; 
Pennlman and Dom1n1ck 1980). He captured both transactional data (what 
happened and at what terminal) and temporal data (when did 1t happen). For 
the purposes of our analysis, we define "transaction" as any set of keystrokes 
followed by a "carriage return" or "enter," and "session" as a sequence of 
transactions bounded by a user start and end time. 

The Library Control System, like many other online catalogs, differs from most 
types of Information retrieval systems 1n that users do not have to logon or 
logoff. They simply walk up to a dedicated LCS terminal and begin entering 
LCS commands. Users neither' Issue a specific "start of session command nor 
Identify themselves to the system. While this feature makes the system easier 
to use and guarantees anonymity. 1t creates a problem for the researcher who 
wishes to delineate the transactions belonging to Individual users. To our 
knowledge, we are the first to attempt an onl1nemon1tor1ng study of user 
behavior where user sessions were defined with data external to the system. 

We collected observational data to establish the session delimiters. The 
advantage of session data over raw transactions 1s that we can study the 
behavior of Individual users, ;ither than just an aggregate of system use. 
Such data allow us to study the types of searches people perform, what 
difficulties they have with the system, and how much time they spend with 1t. 

Given these considerations, we chose to explore the following research 
questions with these data. 

1. How long do people spend at a terminal 1n a session? 

a) when measured 1n elapsed time? 

b) when measured 1n number of transactions? 

2. How does session length vary (1n time and 1n transactions)? 

a) by campus library? 

b) by academic quarter? 

c) by terminal ty^e (sitting or standing)? 

3. How are commands distributed? 

a) by campus library? 

b) by academic quarter? 
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4, How does behavior vary by session? 

a) What 1s the composition or search types (author, title, subject, etc.) 
within a session? 

I) by campus library? 

II) by academic quarter? 

b) What proportion of all sessions contains any subject searching? 

I) by campus library? 

II) by academic quarter? 

c) What proportion of all sessions consists entirely of errors? 

I) by campus library? 

II) by academic quarter? 

5. How does behavior vary by length of session? 

a) 1n terms of command distribution? 

b) 1n terms of the proportion of subject searching? 

c) 1n terms of all-error sessions? 

The data are rich enough to support many further Investigations, but we will 
11m1t our discussion to these research questions for this summary report. 
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4.0 METHODOLOGY 



We combined two methodologies for data collection 1n this study: online 
monitoring of selected terminals and observation of users to determine when 
they began search sessions. We gathered data unobtrusively by roonltorirg all 
transactions from a total of 45 terminals In five of The Ohio State University 
campus libraries. The terminals were monitored for three weeks 1n each of the 
winter, spring, and autumn terms and two weeks of the summer term 1981. 

We utilized observation data which were being collected for an NSF-sponsored 
queuing study of the same terminals (To lie 1982). The use of the observation 
data provided us with a rich dataset at no additional collection cost over 
gathering the monitoring data. 

During the NSF study, observers were posted 1n all terminal areas for two-hour 
time blocks at peak periods of the day during the data collection periods. 
The terminals were monitored during the same periods. Each time a d1ffere.it 
person approached a terminal, the observer noted the exact arrival time and 
the ermlnal Identification number. The observers were performing a queuing 
study requiring only arrival time data, and they did not record the time the 
person left the terminal. 

The observer data were keyed to tape, and we then attempted to match the 
observer times with the transaction times on the monitoring tapes, resulting 
In data parsed Into user sessions. The matching process turned out to be an 
extremely difficult task— one to which we devoted nearly a year of effort. We 
quickly realized that much more 1s Involved than merely matching two sets of 
time stamps. We found that we had to account for several additional 
variables. These variables Included 1) the drift of both the computer time 
clock and the observers' watches (though synchrony had been attempted); 
2) any elapsed time between the observation of the user at the terminal and 
entry of the first carriage return; and 3) missing observations of users (for 
whom we had transactions) or Incorrectly recorded observations of users {for 
whom we had an observation time, but no transactions). After extensive 
analysis, we designed an algorithm which was able to match 61.5X of the user 
transactions vlth sessions. 

To test If the parsing was accurate, we compared the command distribution over 
all the data (prior to parsing) for the month of May 1981 with the co..imand 
distribution for the parsed data for spring term 1981 (Including May) ard 
found that the command distributions were not significantly different 
(p > 0.05) by a ch1-square test. We concluded that the parsed data were an 
accurate sample of the larger dataset on this gross measure, and we proceeded 
with the data analysis. 

We reduced the overall dataset In several steps before obtaining the final 
dataset for analysis. We began with the set of all 66,543 transactions 
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(during 6,045 user sessions) from all LCS terminals during the 11 weeks of 
data collection. From this dataset, we extracted the transactions for the 
public terminals (excluding those In staff areas) In the five campus libraries 
for which we had -*">«*vat1on data. This dataset was further reduced to only 
those hours during «.. ( ch observers were present. We sorted the data Into 
Individual datasets for each terminal and each academic quarter, resulting In 
170 datasets. Me had observed a total of 45 terminals over four academic 
quarters, but no summer term observation was done at 10 of the terminals. 

Of the 170 datasets, we selected 118, which we knew had few, If any, 
observational problems, and matched these against the appropriate observation 
data to match them Into sessions. He were able to parse 61.5X of the 
transactions In the 118 datasets Into unique user sessions. 

He systematically selected 15 of the 45 terminals for final data analysis, 
representing all libraries and all academic quarters, balancing sitting and 
standing terminals and traffic locations in each library. No summer data 
collection was performed at two of the libraries, resulting 1n a loss of four 
of the 60 possible datasets. All of the analysis reported here 1s based on 
the rem. nlng 56 datasets. 

The data used In the analysis In this report are summarized In table 1. 

TABLE 1 
Data Used for Analysis 



Library Number of Number of Number of Number of 

Quarters Terminals Transactions Sessions 



Main 

Info. Desk area 4 5 9,815 610 

Card Catalog area 4 2 16,265 1,391 

Education 4 2 9,595 1,035 

Engineering 3 2 11,366 1,190 

Undergraduate 4 2 10,136 882 

Hest Campus 3 2 9,366 937 

Total 15 66,543 6,045 



Quarter Number of Number of Number of Number of 

Libraries Terminals Transactions Sessions 



Autumn 5 15 21,795 2,029 

Winter 5 15 21,108 1,911 

Spring 5 15 19,480 1,743 

Summer 3 11 4,160 364 

Total 66,543 6,045 
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The data elements collected are summarized 1n table 2. 



TABLE 2 
LCS Data Elements 



Data Collected 

• Online 

1. Terminal Identification number 

2. Commands 

3. Search key 

4. Time of command (to the second) 

5. Date of transaction 

6. Computer response (match, no match, error) 

• Offline 

1. Terminal Identification number 

2. Session start time 

3. Male/female 

4. Staff/patron 

Collected, Hot Used 

• Online 

1. Text of search key 

3 Offline 

1. Male/female 

2. Staff/patron 

Hot Collected 

1. Number of matches 

2. Patron Identification 

3. Session end time 



In order to analyze the temporal data, we noted the time at which each 
transaction occurred and the computed elapsed time between transactions, or 
"gap time." The gap time was defined as the amount of time spent In a 
command state." For example, 1f a user typed 1n a title command, 45 seconds 
later typed 1n a page number command, and 30 seconds after that typed In a 
line number command, that user spent 45 seconds In the "title command state" 
reviewing the results of the title search, followed by 30 seconds In the "page 
command state"; the gap times were 45 seconds and 30 seconds, respectively. 
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He used the session delimiters and gap times to compute two measures of 
session length. Session length 1n number of transactions 1s simply a count of 
all transactions In a session. Session length In elapsed time 1s tho total of 
all the gap times from the first through the last transaction. Note that the 
latter computation does not Include a gap time for the last transaction of the 
session; we do not know the elapsed time from the last carriage return to the 
time the user left the terminal. Therefore , the measure of session length 1n 
elapsed time slightly underestimates the true value. 

Data analysis was performed using the Statistical Analysis System (SAS) and 
custom-designed stochastic process and graph analysis programs. 
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5.0 LCS STRUCTURE 



The Library Control System (LCS) became operational as a combined catalog and 
circulation system In 1970; public access terminals were first Installed In 
1975. In 1978, the online catalog capability was added and now provides 
full-record access to materials acquired after 1973. As of 1982, 115 
dedicated LCS public access terminals were available throughout the OSU 
library system, and nearly 14 mlllf t transactions were performed on the 
system 1n the 1981/82 academic yea- LCS serves a population of 57,000 
students, 18,000 faculty and staff plus an unknown number of library 
visitors, while the card catalog is still available, It Is no longer kept 
current. For additional Information on the development of LCS, refer to the 
reports of Norden and Lawrence (1981) and Miller (1979). 

LCS Is actually both an online catalog and a circulation system. Terminals 
located on library circulation desks and In library staff offices have 
capabilities to check materials In and out and to search patron files, In 
addition to online catalog access. Terminals In public areas have online 
catalog access only. Most of the terminals are of the video display tube 
(V0T) variety; a few printing terminals are available throughout the library 
system. Some dial-up access Is offered as well. 

He collected data only from VDT terminals In the library. Since our objective 
was to capture patron (rather than library staff) behavior, we selected for 
analysis terminals that received heavier patron than staff usage, and we 
Instructed the observers not to record any staff use of the terminals. As 
noted above, such gaps In the collected data complicated the process of 
matching transaction tapes with observer records. This procedure would likely 
be changed 1n any future monitoring effort. 

The Library Control System Is a command-driven system which allows searching 
by author, title, author/title combination, subject heading, and call number. 
No Boolean combinations of these access points are possible, however. To 
search, one types In a three-character command to Indicate the type of search 
(e.g., AUT for author), followed by a slash and a search key containing a 
specified number of characters describing the Item or topic sought (the number 
of characters In the search key varies by search type). The system response 
varies by command, returning a list of one-11ne entries matching the search 
key In most cases. The number of steps required to complete a search varies 
by search type as well. The display of a single record Includes the 
bibliographic description (standard Information found on a catalog card), the 
call number, owning libraries, and the circulation status of each copy 
(currently checked 1n, or, 1f out, when due). The complete 11st of searching 
commands and their structures 1s outlined In table 3. 
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TABLE 3 

Library Control System Taxonomy 



A. LCS search commands can be divided Into three types based on the number 
of steps required to arrive at the display of a bibliographic and 
circulation record. Each step has a unique paging command associated with 
i t» 

a. 3-step command: SIS 

!>• 2-step commands: AUT, ATS, TLS, SPS 

c. 1-step commands: FBC, DSC 

B. Search commands and associated paging commands (user-originated 
commands that allow him/her to page or browse through the material 
retrieved). 



STEP COMMAND 



1. 


SIS 


AUT 


ATS 


TLS 


SPS 


FBC 


DSC 


la. 


PS 


PG 


PG 


PG 


PG 


PD 


PD 


2. 


SBL 


DSL/FBL 


DSL/FBL 


DSL/FBL 


DSL/FBL 






2a. 


PG 


PD 


PD 


PD 


PD 







3. DSL/FBL 
3a. PD 



C. Purpose and Result of Each Command 

COMMAND PURPOSE OF COMMAND (P) AND RESULTING DISPLAY (R) 



3-STEP COMMAND: SIS 

1. SIS P: Search by subject 

R: List of subject headings 

la. PS P: Browse through 11st of subject headings 
R: Next or previous page of subject headings 

2. SBL P: Select a subject heading by line number 

R: List of records matching subject heading selected 

2a. PG P: Browse through list of titles 
R: Next or previous page. of titles 

3. DSL P: Select a specific title by line number 

R: Display of short bibliographic record matching line number 

3. FBL P: Select a specific title by line number; restricted to Items with 
a "full bibliographic record" In file 
R: Display of full bibliographic record matching line number 

3a. PD P: Page through a bibliographic record, short — full, which requires 
more than one screen 
R: neAi or previous page of bibliographic record, short or full 
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TABLE 3— Continued 



AMMAN D PURPOSE OF COMMAND (P) AND RESULTING DISPLAY (R) 

-STEP COMMANDS: AUT, ATS, TLS, SPS 



AUT P: Search by author name 

R: List of records matching author name 

ATS Search by author and title combination 

R: List of records matching author/title combination 

TLS P: Search by title 

R: List of records matching title 

SPS P: Search by general call number (not restricted by Cutter number) 
R: List of records matching call number 

a. PG P: Browse through 11st of titles 
R: Next or previous page of titles 

2. DSL P: Select a specific title by line number 

R: Display of short bibliographic record matching line number 

2. FBL P: Select a specific title by line number; restricted to Items 
with a "full bibliographic record" 1n file 
R: Display of full b1b11ogreph1c record matching line number 

2a. PD P: Page through a bibliographic record, short or full, which requires 
more than one screen 
R: Next or previous page of bibliographic record, short or full 

1-STEP COMMANDS: FBC, DSC 

1. DSC P: Search by exact call number to locate a specific title 

R: Display of short bibliographic record matching call number 

1. FBC P: Search by exact call number to locate a specific title for an 
Item which has a "full bibliographic record" 1n the file 
R: Display of full bibliographic record matching call number 

la. PD P: Page through a bibliographic record, short or full, which requires 
more than one screen 
R: Next or previous page of bibliographic record, short or full 
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6.0 RESULTS 



Results of the analyses performed are presented Individually and are discussed 
further In the discus ion section. 



6.1 Session Length 

We examined session length, both In terms of number of transactions and In 
elapsed time. As described above, session length In number of transactions 1s 
simply a count of commands given by the user during a session; session length 
In elapsed time 1s measured from the first through the last carriage return, 
and, because the elapsed time from the last carriage return to the moment when 
the user leaves the terminal 1s unknown, this measure slightly underestimates 
session time. 

Session length, both In number of transactions and In elapsed time, was found 
to be highly skewed to the left. Most sessions were very short, but a few 
were very long. All the session length distributions, by library, by quarter, 
and by Individual terminal, were different from a normal distribution 
(p < 0.01) by a Kolmogorov-Smlrnov D-stat1st1c test of normality. Due to the 
non-normality of the data, Mode and median are used to describe mosc 
Individual distributions rather than the mean and standard deviation. 



6.1.1 Session Length by Library 

We hypothesized that session length might vary by campus library, due to 
differences in their collections and their users. LCS 1s a union catalog for 
the OSU system; therefore, the search process, database, and results are the 
same no matter where the search Is performed. 

The Education and Engineering libraries are specialized collections for those 
departments; Undergraduate and West Campus libraries both serve a 
predominantly undergraduate population. The Main library contains the primary 
research collection for the University. Main library terminals have been 
divided Into two groups: those In the main Information Desk area of the 
library (labeled Info. Desk) which has the largest cluster of terminals (24) 
on campus and the Card Catalog area (labeled Card Cat.), which has an 
additional 4 terminals. We have treated these as separate libraries 
throughout the study due to differences In the characteristics of use. The 
Education library has 7 terminals, Engineering has 4, and the Undergraduate 
and West Campus libraries each have 6 terminals. The session length data for 
these libraries are- summarized In table 4. 

As can be seen from table 4, the median number of transactions 1s stable 
across libraries, but the elapsed time of sessions 1s not. 
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TABLE 4 
Session Length by Library 



All Four Quarters 



Library 


No. of Transactions 


Elapsed Time (Sec.) 


Number of 




Mode 


Median 


Mode 


neai an 


Sessions Transactions 


Main— Info. Desk 


2 


5 


18 


145 


610 


9,815 


Main— Card Cat. 


1 


7 


101 


236 


1,391 


16,265 


Education 


2 


5 


11 


142 


1,035 


9,595 


Engl neerl ng 


2 


6 


18 


160 


1,190 


11,366 


Undergraduate 


2 


6 


40 


186 


882 


10,136 


West Campus 


2 


6 


49 


235 


937 


9,366 



6.1.2 Session Length by Academic Quarter 

We hypothesized that session length might vary by academic quarter. The data 
are summarized In table 5. 

As stated 1n the Methodology section, the data were collected from January to 
December of 1981, but are arranged by academic year as this 1s the usual 
progression of use. Autumn quarter (81/82 academic year) has the longest 
search time, followed by winter, spring, and summer (80/81 academic year). 
People appear to do the same amount of searching (1n number of transactions) 
each quarter, but get faster 1n the later quarters of the academic year. 

We can see that the patterns of use are much different between LCS and such 
Information retrieval systems as MEDLINE. Pennlman (1981) found that sessions 
averaged 47.5 transactions and 9 minutes and 45 seconds 1n length, far longer 
than LCS sessions. LCS sessions have a median of 5 to 6 transactions and a 
typical elapsed time of 2 to 4 minutes. 



25 



Report Number: 0CLC/0PR/RR-83/7 
Date: 1983 August 31 



TABLE 5 

Session Length by Academic Quarter 



All Libraries 



Quarter 


No. of Transactions 


Elapsed Time (Sec.) 


Number of 


Mode 


neaian 


node 


Median 


Sessions 


Transactions 


Autumn 


2 


6 


18 


189 


2,029 


21.795 


Winter 


2 


6 


20 


182 


l,91i 


21.108 


Spring 


2 


6 


£1 


168 


1.743 


19.480 


Summer 


2 


5 


42 


143 


364 


4.160 



6.1.3 Session Length by Terminal Type 

He suspected that users might spend more time at those terminals with a chair 
or stool 1n front of them than at terminals without seating. We compared 
sitting and standing terminals 1n each library. Note that the Card Catalog 
area of the Main library has no sitting terminals and that West Campus library 
has no standing terminals. An additional category of "fast" was created for 
the Information Desk area to identify those terminals near the entrance to the 
area which appeared to have more rapid search sessions. These so-called 
fast" terminals are all standing terminals. Results are summarized 1n 
table 6. 

Session length 1s greater 1n all cases, both 1n elapsed time and 1n number of 
transactions, for sitting than for standing terminals. Session length varies 
considerably among libraries. 
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TABLE 6 

Session Length, Sitting v«. Standing Terminals 



Info. Desk Card Cat. Education 

S1t ^tanT"" Fast Stand Sit Stand 

TIME (Sec.) 

Median 408.5 185 149 236 172.5 121 

Mode 49 45 18 101 11 45 

No. of Sessions* 522 1,025 634 1,243 430 511 

TRANSACTIONS 

Median 11 7 5 7 6 5 

Mode 5 2 2 1 2 2 

No. of Sessions* 550 1,096 696 1,387 461 575 



Engl neerl ng 
>7t Sland 



TIME (Sec.) 

Median 213.5 13. .5 

Mode 18 13 

No. of Sessions* 426 660 

TRANSACTIONS 

Median 6 5 

Mode 2 2 

No. of Sessions* 458 728 



West Campus 
— Sit 



Undergraduate 
5Tt Stand 



235 
49 
861 



231 
33 
529 



106.5 
95 
280 



6 
2 

935 



8 
4 

562 



5 

2 

319 



*The number of sessions 1s fewer for computations by elapsed time than for 
computations by number of transactions, because we have no elapsed time 
associated with the last transaction In the session. Therefore, 
single-transaction sessions have no time associated with them and are not 
Included In the computation. 



6.2 Command Distribution 

We wished to examine an aggregate of command usage across libraries and across 
quarters. (See table 3 for a description of command structure.) 

Command distribution by library and b: quarter 1s shown In tables 7 and 8. 
Two types of errors are defined. A "typing error" 1s a command unrecognized 
by the system, presumably some typographic* 1 error 1n the three-character 
command code. Entries In which the command was correct, but some detectable 
error was made 1n the search key, are considered "logical errors." 
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TABLE 7 
Command Usage by Library 







Percent of 


All Commands 






Command 


Info. 


Card 


Educ. 


Eng. 


West 


Under- 


No of 




Desk 


Cat. 




Campus 


grad. 


Transactions 


Search Commands 














AUT 


7.2 


6 7 


6 6 

v • V 


a 9 




4.9 


3,802 


TLS 


10,4 


7 6 


12 2 


10 5 


5 6 


8.9 


6,047 


ATS 


4.6 


2.5 


4.3 


2.4 


8.5 


5 6 

V # V 


2 906 

b , + VV 


SIS 


8.2 


13.3 


7.6 


10.7 


11.8 


9.1 


6,951 


Call Number 
















DSC 


4.3 


4.3 


4.9 


4.5 


1.8 


2.5 


2,529 


FBC 


0.1 


0.4 


0.3 


0.1 


0.3 


0.1 


147 


SPS 


0.8 


0.5 


0.6 


0.7 


0.6 


2.7 


641 


Subtotal 


(35.6) 


(35.3) 


(36.5) 


(33.8) 


(31.9) 


(33.8) 


(23.023) 


Paging Commands 














PD 


2.1 


1.5 


3.3 


2.1 


1.4 


1 Q 

1.9 


1 oco 
1 ,J5u 


PG 


14.4 


11.1 


14.1 


14.0 


10.9 


13.2 


8,525 


PS 


4.4 


6.4 


3.1 


6.0 


6.5 


4.7 


3,541 


Subtotal 


(20.9) 


(19.0) 


(20.5) 


(22.1) 


(18.8) 


(19.8) 


(13,416) 


Line Number Commands 














DSL 


22.5 


18.0 


24.4 


20.8 


22.8 


14. 7 


14,473 


FBL 


1.6 


1.7 


1.3 


1.6 


1.2 


1.6 


1,018 


SBL 


7.8 


10.5 


6.1 


9.6 


8.8 


8.0 


5,791 


Subtotal 


(31.9) 


(30.2) 


(31.8) 


(32.0) 


(32.8) 


(34.2) 


(21,282) 


Errors 
















Typ Err 


5.9 


7.3 


5.9 


5.3 


8.2 


6.1 


4,326 


Log Err 


5.6 


8.1 


5.3 


6.7 


8.0 


5.9 


4,498 


Subtotal 


(11.5) 


(15.4) 


(11.2) 


(12.0) 


(16.2) 


(12.0/ 


(8.824) 


Total 


100.0 


100.0 


100.0 


100.0 


100.0 


100.0 


66,545 



The proportion of commands across parts of the search *s relatively stable 
across libraries, but there 1s a«ton$1derable difference among libraries 
(table 7). For example, far more subject searches are performed at terminals 
1n the Card Catalog area of the Main library than 1n the Information Desk area 
of the same library. Subject searching 1s next highest at the two libraries 
with a predominantly undergraduate population— West Campus and Undergraduate 
libraries. West Campus, where most undergraduate students perform required 
searches for their Introductory library use course, has about half as many 
call number searches and twice as many author/title searches as the rest of 
the libraries; the number of errors 1s also highest at West Campus. 
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Use of commands 1s stable over the autumn, winter, and spring terms, but 1t 
shifts 1n the summer, particularly 1n the distribution of searching commands 
(table 8). 



TABLE 8 



Command Usage by Quarter 







Percent of 


Commands 
















NiJfnhpr of 


Command 


Autumn 
nil LUIIIN 


Winter 


o\ii i iiy 


a limner 


Tf» ncjif*+4ftnc 


Search Commands 












AUT 


4.8 


5.6 


6.0 


9.2 


3,802 




O »D 




Q 1 




O ,U4/ 


TC 
* 1 J 


A 7 


o.4 


o»l 


3.3 


o one 


etc 


in i 


ii i 

li.i 


ii n 
11*0 


6.4 


£ Aft 

0,951 


ta 1 1 NumDer 










DSC 


3.0 


4.6 


3.9 


3.9 


2,529 


FBC 


0.1 


0.3 


0.4 


0.2 


147 


Or d 


n q 

U.J 




1 7 




041 


Subtotal 


(34 1) 






MS ft) 




Paging Commands 












PO 


1.9 


2.2 


1.9 


2.3 


1,350 


PG 


14.3 


12.3 


10.8 


17.1 


8,525 


PS 


5.7 


5.7 


4.7 


4.2 


3,541 


subtotal 


(21.9) 


(20.2) 


(17.4) 


(23.7) 


(13,416) 


Line Number Commands 












DSL 


22.0 


20.6 


22.3 


23.9 


14,473 


FBL 


1.6 


1.4 


1.6 


1.0 


1,018 


SBL 


9.1 


8.5 


9.1 


5.8 


5,791 


Subtotal 


(32.7) 


(30.5) 


(33.0) 


(30.7) 


(21,282) 


Errors 












Typ Error 


6.1 


6.7 


7.0 


5.4 


4,326 


Log Error 


5.2 


8.1 


7.6 


4.3 


4,498 


Subtotal 


(11.3) 


(14.8) 


(14.6) 


(9.7) 


(8.824) 


Total 


100.0 


100.0 


100.0 


100.0 


66,545 



6,3 Analysis by Session 

The partition of transaction data Into sessions allows us to look at the 
behavior of Individual users 1n searching the database. We were particularly 
Interested 1n whether users tend to perform only one type of search 1n a 
session (and 1f so, which type) or multiple search types 1n a session. Also 
of Interest 1s the proportion of sessions 1n which any subject searching 
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occurs* Such data have an Important Impact on design and training, and the 
Information has not been available from other types of catalog use studies* 
These data and data on error rates are summarized 1n the following sections* 



6*3*1 Session Search Types by Library 

In table 9, we summarize the distribution of searches within sessions* Search 
types are listed by search command (see table 3 for a description of the 
commands)* An "AUT" session, for example, contains only author searches 
(paging and display commands and errors were Ignored 1n the computation)* A 
"MIXED session contains more than one type of search, and a "NONE" session 
contains no search commands (these are probably error-laden sessions)* 

Note that about one-third of all sessions contain multiple types of searches. 
The distribution of session search types varies considerably among libraries* 
The proportion of mixed sessions, for example, 1s much higher 1n both areas of 
the Main library than 1n the departmental libraries* 

TABLE 9 

Session Search Types by Library 



Command 


Info. 


Desk 


Card Cat. 


Education 




Sessions 


Sessions 


Sessl 


ons 




No. of 


X of 


No. of 


X of 


No. of 


X of 


AUT 


78 


12.8 


144 


10.3 


95 


9.2 


ATS 


21 


3.4 


55 


3.9 


63 


6.1 


TLS 


99 


16.2 


175 


12.6 


279 


26.9 


SIS 


115 


18.8 


316 


22.7 


123 


11.9 


SPS 


1 


0.2 


9 


0.6 


6 


0.6 


DSC 


17 


2.8 


73 


5.2 


90 


8.9 


FBC 


0 


0.0 


4 


0.3 


0 


0.0 


MIXED 


250 


40.9 


526 


37.8 


351 


33.9 


NONE 


39 


4.9 


91 


6.5 


29 


2.8 


Total 


611 


100.0 


1,393 


100.0 


1,036 


100.0 



Engineering Undergraduate West Campus Total 
Sessions Sessions Sessions Sessions 





No. of 


X of 


No. of 


X of 


No. of 


X of 


No. of 


X of 


AUT 


74 


6.2 


63 


7.1 


36 


3.8 


490 


8.1 


ATS 


56 


4.7 


92 


10.4 


295 


31.4 


582 


9.6 


TLS 


311 


26.1 


145 


16.9 


89 


9.5 


1,102 


18.2 


SIS 


224 


18.8 


173 


19.6 


191 


20.4 


1,142 


18.9 


SPS 


12 


1.0 


5 


0.6 


5 


0.5 


38 


0.6 


DSC 


84 


7.0 


56 


6.3 


14 


1.5 


334 


5.5 


FBC 


0 


0.0 


0 


0.0 


0 


0.0 


4 


0.0 


MIXED 


368 


30.9 


315 


35.7 


260 


27.7 


2.070 


34.2 


NONE 


62 


5.2 


30 


3.4 


48 


5.1 


290 


4.8 


Total 


1,191 


100.0 


883 


100.0 


938 


100.0 


6,052 


100.0 
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6.3.2 Session Search Types by Quarter 

In table 10, we present the analyses of session search types by quarter. 

TABLE 10 
Session Search Types by Quarter 

Command Autumn Winter Spring Summer 

Sessions Sessions Sessions Sessions 





No. of 


% of 


No. of 


% of 


No. of 


X of 


No. of 


* of 


AUT 


114 


5.6 


158 


8.3 


175 


10.0 


43 


11.8 


ATS 


342 


16.8 


13C 


6.8 


90 


5.2 


20 


5.5 


TLS 


305 


15.0 


375 


19.6 


358 


20.5 


62 


17.0 


SIS 


377 


18.6 


370 


19.3 


347 


19.9 


48 


13.2 


SPS 


18 


0.9 


9 


0.5 


11 


0.6 


0 


0.0 


OSC 


85 


4.2 


139 


7.3 


79 


4.5 


31 


8.5 


FBC 


1 


0.0 


2 


0.1 


1 


0.0 


0 


0.0 


MIXED 


695 


34.2 


640 


33.4 


593 


34.0 


142 


38.9 


NONE 


93 


4.6 


90 


4.7 


90 


5.2 


19 


5.2 


Total 


2,030 


100.0 


1,913 


100.0 


1,744 


100.0 


365 


100.0 



The distribution of session search types 1s more stable when viewed by quarter 
than when viewed by library. As 1n other measures (see tables 5 and 8), 
summer quarter data are least like that of other quarters. During the summer 
quarter the composition of the student body 1s very different from the other 
three quarters. There 1s a much lower percentage of undergraduate students 
and a higher percentage of graduate students and continuing education 
students. Thus, the search patterns cf graduate students, faculty, and 
library employees using public terminals have more of an Impact on LCS 
searching patterns during the summer quarter than during the other three 
quarters. The summer term dataset also 1s a much smaller one than for other 
quarters. Campus enrollment 1s lower In the summer term; no data were 
collected In the Engineering or West Campus libraries during the summer 
quarter (see table 1), and we collected three weeks of data during autumn, 
winter, and spring quarters, but only two weeks' data In the summer quarter. 



6.4 Subject Searching Sessions 

Librarians have long been Interested in the proportion of catalog searching 
for known Items vs. the proportion of subject searching, so that they might 
allocate cataloging and reference resources more accurately. While many 
online catalogs (Including LCS) provide subject access only by standard 
Library of Congress Subject Heading (LCSH), effective subject access requires 
keyword searching 1n title and subject fields, cross references, and 8oo1ean 
logic capabilities. These are expensive additions to a system and must be 
justified by demand. 
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Monitoring analysis allows us to determine not only the proportion of searches 
which are subject requests and the proportion of sessions which are 
subject-only searches, but also the proportion of sessions 1n which any 
subject search t jg occurs. The latter measure may most accurately reflect the 
demand for s bject searching. The data are summarized by library 1n table 11 
and by quarter 1n table 12 f respectively. 

These data show that roughly one-third of all sessions Include at least one 
subject search command, and that subject searching 1s much lower during the 
summer term. 



TABLE 11 

Sessions with Subject Commands (SIS) by Library 



Percent of All Sessions 



Info. Desk Card Cat. Education Engineering Undergraduate West Campus 



36.4 41.0 21.6 32.8 35.6 35.8 



TABLE 12 

Sessions with Subject Commands (SIS) by Quarter 

Percent of All Sessions 
Autumn Winter Spring Summer 

34.0 35.4 34.5 23.9 
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6.4.1 All-Error Sessions 

In the monitoring analysis of MEDLINE, Pennlman (1981) found that users are 
frequently unsuccessful 1n logging onto the system* We suspected that similar 
events occur 1n LCS, where users attempt to search, make a series of errors, 
and abandon the system without having any commands accepted. To test this 
hypothesis, we examined sessions which consisted only of errors* Tables 13 
and 14 summarize the data by library and by quarter, respectively* 



TABLE 13 
All-Error Sessions by Library 



Percent of All Sessions 



Info. Desk Card Cat, Education Engineering Undergraduate West Campus 



10.0 15.4 11.5 11.8 10.9 il.7 



TABLE 14 
All-Error Sessions by Qiarter 



Percent of All Sessions 



Autumn W1 nter Spr1 ng Summer 



11.5 13.4 11.8 12.1 



6.5 Analysis of Sessions by Length 

The session length distribution 1s highly skewed to the left, which means that 
most sessions are very short. We suspected that there might be a variety of 
different characteristics between short and long sessions, particularly 1n 
distribution of search type and 1n error rate. 

To examine these hypotheses, we selected four terminals with similar session 
lengths and command distribution characteristics and combined them Into one 
dataset. He then split the dataset Into four separate datasets at three 
session length points (In number of transactions): mode, median, and third 
quartlle. The four groups are "Short" (sessions with length less than or 
equal to the mode), "Medium" (length greater than the mode, but less than or 
equal to the median), "Medium-Long" (greater than the median, but less than or 
equal to the third quartlle), and "Long" (greater than the third quartlle). 
The datasets are described In table 15. 
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TABLE 15 

Distribution of Session Length Data 





Number of 


Number of 


Lenjth 


Transactions 


Sessions 


Short 


324 


203 


Medium 


917 


236 


Medium-Long 


1,313 


167 


Long 


5,074 


210 


Total 


7,628 


816 



Each of these datasets was analyzed separately, and the results are displayed 
In tables 16 through 18. 



These data show considerable differences among sessions of different lengths. 
The proportion of commands devoted to searching drops steadily (56.5S to 
30. IX) from short through long sessions. The proportion of author, title, and 
author/title searches drops, while the proportion of subject and call number 
searches Increases. The proportion of sessions with any subject searching 
Increases dramatically from 1 .o% to 62 .4X. 

The proportion of typing errors drops from short to long sessions, but the 
overall proportion of errors does not drop until the long sessions. Almost 
half (43. 3X) of the short sessions (those less than or equal to the mode) 
consist entirely of errors, while the longer sessions are rarely all-error 
sessions. 



We wished to compare command sequences to determine If patterns varied between 
short and long sessions; that Is, are long sessions unique, or are they just a 
series of short sessions? To compare these different command sequences, we 
treated Individual commands as states In a Markov chain. A two-sample 
Kolmogorov-Smlraov (KS) test was applied to determine differences 1n 
transitions between command states at the zero order (cumulative frequency), 
first order (one-step transitions, or two consecutive states), and second 
order (two-step transitions, or three consecutive states). Six KS tests were 
performed: short sessions vs. long and short plus medium vs. long at each of 
zero-, first-, and second-order transitions. All tests were significant 
(p < 0.001), suggesting that patterns of use vary greatly by session length. 
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TABLE 16 
Command Usage by Session Length 



Percent of Commands 


Command 










Number of 


Short 


Medium 


Med-Long 


Long 


Transacti on 


Search Commands 












A NT 

AUT 


11 A 


i i ^ 

11.7 


7.5 


3.9 


441 


Tl C 


on a 
2Q.4 


1 A C 

14.5 


1 O A 

13.0 


6.7 


711 


ITC 


6.5 


^ i 
7.1 


4.6 


1.9 


241 


SIS 


8.3 


7.6 


9.4 


13.2 


892 


Call Number 












DSC 


9.3 


C 1 

5*1 


5.2 


ft M 

3.4 


316 


FBC 


0.0 


n i 

U.l 


n n 
U.U 


U.U 


2 


SPS 


0.6 


1 0 

1 *c 


i i 
1.1 


1.1 


ft 1 

81 


Subtotal 


(56.5) 


147.0 J 


140.8) 


(30.2) 


i ft f ft M \ 

(2 t 684) 


Paging Commands 












PD 


1.2 


2.6 


1.9 


1.6 


136 


PG 


1.5 


C 1 
0 .1 


11 Q 


19.2 


1 1 OO 

1,182 


PS 


0.6 


3.4 


4.3 


8.4 


517 


Subtotal 


(3.3) 


(11.1) 


(18.0) 


(29.2) 


(1,835) 


Line Number Commands 










DSL 


23.2 


97 Q 


tU.O 


1 O 1 

lo.l 


1,4/3 


FBL 


0.0 


0.9 


1.6 


1.7 


116 


SBL 


3.4 


3.8 


6.2 


11.1 


689 


Subtotal 


(26.6) 


(28.6) 


(28.1) 


(30.9) 


(2.284) 


Errors 












Typ Error 


8.3 


7.7 


5.5 


5.1 


429 


Log Error 


5.3 


5.2 


7.8 


4.5 


396 


Subtotal 


(13.6) 


(12.9) 


(13.3) 


(9.6) 


(825) 


Total 


100.0 


100.0 


100.0 


100.0 


7,628 
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TABLE 17 

All-Error Sessions by Session Length 

Percent of All Sessions 
Short Medium Medium-Long Long 

43.3 3.8 1.8 0.0 



TABLE 18 

Sessions with Subject Commands by Session Length 



Percent of All Sessions 


Short 


Medium Medium-Long 


Long 


12.8 


19.5 34.7 


62.4 



ERIC 



36 



25 



Report Number: 0CLC/0PR/RR-83/7 
Date: 1983 August 31 



7.0 DISCUSSION 



The data presented 1n this report characterize the use of one online catalog 
system, at one major university library, at one point 1n time. Until 
comparable studies are performed on other systems, we will not know whether 
these data are representative of user online catalog behavior 1n general. 
However, due to vast differences between systems and library environments, we 
suspect that these data are not representative, except perhaps at the most 
general level, such as for types of searches performed. 

Several Interesting findings emerge from the data presented here. The most 
obvious are the user behavior differences among campus libraries, academic 
quarters, and types of terminal settings. For example, behavior 1s quite 
different between the Information Desk area and the Card Catalog area of the 
same library. He do not know precisely to what this difference may be 
attributed. The Information Desk rmlnals are easily accessible, while a 
user must make a more deliberate e< fort to seek out a Card Catalog area 
terminal. The Card Catalog area terminals also afford more privacy than the 
Information Desk terminals. We expected the Card Catalog area to have a high 
proportion of known Item searches, both from people searching bibliographies 
and from people matching catalog cards to the system to check for 
availability. However, this was not the case— subject searching 1s higher 1n 
the Card Catalog area than for any other library! 

Searching behavior 1n the West Campus and Undergraduate libraries, which are 
populated predominantly by undergraduate students, 1s much different from that 
1n the other campus libraries. Undergraduates take a required course 1n 
library usage which Includes LCS Instruction. Most take the course during the 
autumn quarter, and they have to perform a required set of searches 1n 
conjunction with the course. The students are trained to use the author/title 
search (ATS), and this 1s reflected 1n the command distribution— 31. 41 of West 
Campus searches are ATS only. ATS use 1s high overall during the autumn 
quarter, reflecting the effect of undergraduates running required searches. 
We suspect that the skewed ATS use 1s a training effect, suggesting that we 
might compare 1t to training received by users 1n other libraries. 

Patterns also change 1n command distribution and error rate across quarters. 
Session lengths become continuously shorter from autumn through summer terms, 
suggesting that users are perhaps getting faster as their expertise Increases. 
However, this finding 1s contradicted by the fact that error rates Increase 
for winter and spring terms over those of autumn, then drop off during summer 
term. The Ohio State University LCS user population 1s dynamic, and since we 
do not know that the same users are on the system from autumn through spring, 
little can be assumed 1n terms of transaction effects. In fact, because these 
data were collected 1n one calendar year (1981) rather than 1n one academic 
year, the autumn quarter analyzed 1s for the academic year following that of 
the winter through summer quarters. All we really know 1s that autumn quarter 
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sessions are longer than sessions 1n other quarters. We also know that a 
different population 1s resident during summer, and these patterns are much 
different. 

ic found that 1n each library sessions at sitting terminals were consistently 
longer than sessions at standing terminals, when measured either 1n number of 
transactions or 1n elapsed time. What we do not know 1s which direction 1s 
c*usa1— do people, expecting to search longer or slower, seek sitting 
terminals, or do people just stay longer 1f you give them a stool? The data 
suggest the latter, because session length 1n the Card Catalog area of the 
Main library, which has standing terminals only, 1s very similar to session 
length for standing terminals 1n the Information Desk area; and session length 
at West Campus, which has sitting terminals only, 1s more similar to session 
length at sitting terminals 1n the other departmental libraries than to 
session length at standing terminals. 

• 

The differences between libraries, quarters, and Individual terminals are 
sufficiently Urge that we believe 1t 1s necessary to work with a large 
dataset, such as we have, to get an accurate picture of system usage. In this 
dataset, any random sample from one library or one quarter, much less from one 
terminal, would not be an accurate representation of user behavior. 

While we did not dwell on error behavior 1n this report, we did find ;hat an 
average of 13.3X of all commands are either typographical or logical errors, 
and 12.2X of all sessions consist entirely of errors. However, most of the 
all-error sessions are very short. We do not know 1f these error rates are 
high or low, as we have nothing with which to compare them. Error behavior 
will be explored further 1n a future report. 

The monitoring study of the Library Control System resulted 1n far more data 
than are presented here. We expect to follow this report with several other 
papers discussing 1n more detail such aspects as temporal patterns, error 
patterns, command sequences, search success, and variance by campus library 
and academic quarter. We will also compare the data from LCS with that of 
other online catalogs collected 1n conjunction with the Online Public Access 
Project (Tolle 1983). In addition, we hope to compare these data with users 1 
self-evaluation of searching behavior and satisfaction with LCS as reported 1n 
the user surve" portion of the Online Public Access Project (Markey 1983). 

Because the Library Control System 1s under continual development, we also 
hope to gather more data at a later date. By comparing datasets from two 
points 1n time, we may Identify the effects of system enhancements on user 
behavior. 
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